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Abstract —Multipath TCP (MPTCP) is a transport layer pro¬ 
tocol that allows network devices to transfer data over multiple 
concurrent paths, and hence, utilizes the network resources more 
effectively than does the traditional single-path TCP. However, 
as a reliable protocol, MPTCP still needs to deliver data packets 
(to the upper application) at the receiver in the same order they 
are transmitted at the sender. The out-of-order packet problem 
becomes more severe for MPTCP due to the heterogeneous nature 
of delay and bandwidth of each path. In this paper, we propose 
the forward-delay-based packet scheduling (FDPS) algorithm for 
MPTCP to address that problem. The main idea is that the 
sender dispatches packets via concurrent paths according to 
their estimated forward delay and throughput differences. Via 
simulations with various network conditions, the results show that 
our algorithm significantly maintains in-order arrival packets at 
the receiver compared with several previous algorithms. 

1. Introduction 

It is common nowadays that the network devices are 
equipped with multiple access interfaces, and hence, will po¬ 
tentially have multiple concurrent paths through the Internet’s 
infrastructure. For instances, a personal computer may have 
both Ethernet and Wi-Fi interfaces. Similarly, a smart phone 
usually has multiple radio interfaces, such as cellular (3G/4G), 
Wi-Fi, Bluetooth, etc. On the other hand, the traditional TCP 
protocol only assumes one data flow through the network, and 
hence, may not utilize the network resources efficiently. To 
overcome that, the Internet Engineering Task Force (IETF) 
has standardized the Multipath TCP (MPTCP) protocol (see 
Q-Q) as a major extension of the single-path TCP protocol. 

It allows simultaneous data transfer between end-to-end hosts 
across different paths, and hence, has the capability to increase 
the end-to-end goodput. 

In order to utilize multiple paths in MPTCP, a packet 
scheduler is required; its responsibility is to decide which 
packet to be dispatched via which path. On the other hand, as 
a reliable protocol, MPTCP still needs to deliver data packets 
(to the upper application) at the receiver in the same order they 
are transmitted at the sender. Due to the heterogeneous nature 
of delay and bandwidth of each path, data packets may arrive 
out-of-order at the receiver and have to be re-ordered. Thus, a 
well designed packet scheduling algorithm will lead to a less 
re-ordering effort (both buffer-wise and computation-wise) at 
the receiver, and prevent performance degradation of MPTCP. 

In this work, we propose the forward-delay-based packet 
scheduling (FDPS) algorithm for MPTCP to address the 
problem of out-of-order received packets described above. 
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In particular, our main contributions can be summarized as 
follows: 

• We present a technique to calculate the differences in 
forward delays between paths that does not require 
clock synchronization between source host and desti¬ 
nation host. This technique is of interest on its own, 
and may be useful to other future developments related 
to MPTCP. 

• Based on the above technique, we propose our FDPS 
algorithm to address the problem of out-of-order re¬ 
ceived packets. The main idea is to schedule packets 
according to the estimated delay and bandwidth dif¬ 
ferences between paths in the forward direction. 

• We implement and evaluate our proposed algorithm in 
ns-2 0 and via the Reorder Buffer-occupancy Density 
(RBD) metric, respectively. Our extensive simulation 
results (under various network conditions) show that 
our algorithm achieve the more fraction of in-order 
received packets than do the previous algorithms. 

The remainder of the paper is organized as follows. We 
review the previous works related to MPTCP and packet 
scheduling in Section |I^ In Section we describe our pro¬ 
posed FDPS algorithm for MPTCP, and present performance 
evaluation results in Section IlYl We then conclude our work 
in Section Ivl 

II. Related Work 

Various congestion control algorithms have been proposed 
for MPTCP, e.g., 0,0(8) to take advantage of multiple con¬ 
current paths. In general, the MPTCP congestion control will 
send more data through the paths that have larger congestion 
window size and/or smaller round trip time (RTT). However, 
as mentioned in the Introduction section, the differences of 
bandwidth and delay between paths may cause out-of-order 
packets at the receiver. For instance, it has been shown 0 that 
if two paths have a large difference in bandwidth and delay, 
MPTCP on these paths without a proper packet scheduling 
algorithm will be less effective than traditional single-path 
TCP. 

Recently, several packet scheduling solutions have been 
proposed to address the out-of-order packet problem in 
MPTCP. For example, the multipath transmission control 
scheme (MTCS) |T0[ attempts to combine congestion control 
and scheduling to send in-order packets. The scheme uses a 
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Fig. 1: Forward and backward delays 



load sharing model to distinguish packet loss due to network 
congestion from the one due to wireless channel condition to 
find the optimal path. At the same time, it employs a feedback- 
control-based packet scheduling mechanism to maximize the 
number of data packet sent to the receiver in a timely manner 
without losing their order. Kim et al.’s algorithm GD (called 
Kim-2012 onward in this paper) performs in-order packet 
scheduling by dispatching packets on each path based on RTT 
after successfully sending a data packet and receiving acknowl¬ 
edgment (ACK) for that packet. However, the algorithm uses 
RTT/2 as an estimate for the forward delay, and as illustrated 
in Fig.l^it may not be a good estimation due to the asymmetric 
delay nature of forward and backward paths. 

Although our focus is to estimate the forward delay differ¬ 
ence between concurrent paths and not the forward delay itself, 
we still would like to briefly survey the literature on one-way 
delay estimation without clock synchronization. Tsuru et al. 
eg have proposed a method to estimate clock offset (between 
two hosts) for the case in which the forward and backward 
paths have different bandwidths, but they still assume symmet¬ 
ric propagation and transmission delays on both paths. On the 
other hand, the method proposed by Choi and Yoo (m does 
not require any symmetric or synchronization assumption, but 
it requires the RTT measurements at both sender and receiver, 
and its accuracy strongly depends on the delays of first packets 
in both directions. 

III. Forward Delay-based Packet Scheduling 
(FDPS) Algorithm 

In this section, we present the details of our proposed 
forward-delay-based packet scheduling (FDPS) algorithm. The 
algorithm consists of two parts: i) estimating the forward delay 
differences between paths, and ii) selecting data to send via a 
path when the congestion window is available on that path. 

A. Estimating forward delay differences 

Forward delay is defined as the delay in forward direction 
from sender to receiver. Similarly, backward delay is the delay 
in backward direction (from receiver back to sender), and the 
total of forward and backward delays is the round-trip time 
(RTT) (see Fig. for an illustration). The problem of estimat¬ 
ing forward delay on each path is not trivial in practice because 
the clocks at sender and receiver are often not synchronized. 
Nevertheless, in this section, we show that it is possible to 
estimate the forward delay differences between paths even 
without clock synchronization of sender and receiver. 


Fig. 2: Calculating the forward delay difference 


Our technique is illustrated in Fig. in which the sender 
(S) sends two packets via two different paths to the receiver 
(R). In particular, at time i, S sends packet 1 via Path 1 to 
R, and that packet reaches R at time t^ ^. At time ^2 2 ’ ^ sends 
packet 2 via Path 2 to R, and that packet reaches R at time 
^2 2 - We define the sending time difference As between two 
packets as: 

= As e R, (1) 

and similarly, the receiving time difference Ar between two 
packets: 

Ar = ^ 2,2 - * 1 , 1 ’ ^ 

Let AT be the difference in clock time between S and R, i.e., 
AT = elks — clkT, at G R. (AT is possibly unknown to 
both S and R.) Then the forward delay of packet 1 on Path 1 
is: 

^ 1,1 = - tyi + at. (3) 

Similarly, the forward delay of packet 2 on Path 2 is: 

^ 2,2 = - 4,2 + at. (4) 

Therefore, the forward delay difference between Path 1 and 
Path 2 is estimated as: 

Api,P2 = ^ 1,1 “ ^ 2,2 

= t^,i-Ci,i+AT-t^,2+^2,2-AT 
= ^2,2-^M-^2,2+^M 

= As — Ar. (5) 

Note that the sender S is able to carry out the above estimation 
after receiving ACKs of both packets from both paths. The 
sending timestamps tf j can be put into the data packet option 
and then piggybacked to the sender via the ACK packet option, 
while the receiving timestamps can be put into the ACK 
packet option. 

B. Scheduling data packets 

In order to schedule data packets, we first need to find 
the path with shortest forward delay. Particularly, let V = 
{Pi, P 2 , • • • , Pat} be the set of MPTCP concurrent paths, then 
the path with shorted forward delay Pp can be determined by 

Algorithm 








Algorithm 1 Finding shortest forward-delay path 

Input: a set of paths V = {Pi , P 2 , • " 5 Pn} 
Output: Pi* as the path with shortest forward-delay 

for i = 1 to do 
NegCount[i] ^ 0 

end for 

for all Pi, Pj G P, i 7^ j do 

Calculate ^Pi,Pj using Equation ® 

if Ap. < 0 then 

NegCount[i] ^ NegCount[i] + 1 
else if Ap. p > 0 then 

NegCountfj] ^ NegCount[j] + 1 
else 

Generate a Bernoulli sample s G {0,1} 

if s = 0 then 

NegCount[i] ^ NegCount[i] -f 1 
else 

NegCount[j] ^ NegCount[j] + 1 

end if 
end if 
end for 

/* Pi* is the path with maximum NegCount */ 

P := arg max NegCount[i] 


The main idea of Algorithmis fairly simple: the forward 
delay difference Ap.^^p. between Pp and any other Pj G V 
has to be less than or equal to zero. Thus, the algorithm does 
the comparison for all pairs of paths, and for each path it 
keeps track of the number of times when the difference is 
negative for that path (stored in the NegCount variable, with 
random tie-breaking). Then Pp is the path with largest value 
of NegCount. 

After the path with shortest forward delay is determined, 
the data packets will be scheduled. Suppose that the MPTCP 
sender uses a sending buffer shared among sub-flows, and data 
coming from the application layer is enough to All in the buffer. 
Let the data in the sending buffer be partitioned into packets, 
each packet has the size equal to the MSS (message segment 
size) of TCP paths. Then the scheduler chooses data packets 
with lower sequence numbers to send on paths with smaller 
forward delays. In particular, when the sub-flow on path i 
requires data to transfer, the scheduler will choose data packet 
at index idxi in the sending buffer according to the following 
formula: ^ ^ 

idxi = Ap^^p., X (6) 

where Xi is the average throughput (byte/s) path i. Note that 
path P always picks the data packet at index 0. After one data 
packet is sent, the other packets with higher indices will be 
shifted to All up the buffer. An illustration of the algorithm is 
presented in Fig. 

We would like to emphasize that our packet scheduling 
algorithm is a Pull mechanism (as opposed to Push or Hybrid 
push-pull mechanisms 0). That is, generated data packets are 
stored in the sending buffer and allocated to the sub-flows only 
when the sub-flows have an open window to transmit data (e.g., 
after receiving an ACK). 

We conclude this section with some discussions about the 
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Fig. 3: Dispatching packets based on forward delay differences 


intuition behind our proposed algorithm. One can see that 
xp /MSS is the normalized rate (in packet per second) of 
path i*, and hence, the product Ap.^p.^ x /MSS may 
be interpreted as the “bandwidth-delay product difference” 
between paths i and i*. Therefore, there is roughly enough 
room on path P to send Ap.^p.^ x /MSS packets before 
getting to path i, and thus, the packet to send on path i should 
be right after those packets, i.e., at the index presented in 
Equation 

IV. Performance Evaluations 

In this section, we investigate the performance of our EDPS 
algorithm and compare it with the performances of MTCS 
|T^ , Kim-2012 and EIEO (flrst-in-flrst-out, default in 
MPTCP) in terms of Reorder Buffer-occupancy Density (RBD) 
and Reorder Density (RD) metrics |T^ , both are measured at 
the receiver. By deflnition, RBD is the normalized histogram 
of the occupancy of a hypothetical buffer (at the receiver) 
needed to re-sort out-of-order packets, while RD is deflned as 
the distribution of displacements of packets from their original 
positions, normalized with respect to the number of packets. 
Note that RBD[0] (the value of RBD at index 0) represents the 
density of in-order packets that arrive at the receiver and can be 
delivered to upper application immediately without buffering. 

We would like to point out that although throughput (or 
goodput) is a popular performance metric to benchmark TCP- 
related algorithms, we do not consider goodput here. The rea¬ 
son is that, beside the reordering aspect, goodput also depends 
on many other factors such as data loss or retransmission 
mechanism. On the other hand, both RBD and RD capture 
the essential nature of the reordering problem, and hence, are 
chosen as our main performance metrics. 

Our evaluations are based on topologies shown in Pig. 
g and simulated on NS-2.34 Q. Simulation parameters are 
given in Table |T| The simulation experiments are carried out 
under various network conditions as presented in the following 
subsections. 

A. Two-path MPTCP without background traffic 

In this section, we consider the topology shown in Pig. 
Qa) in which a MPTCP flow runs on two separate paths 
without background traffic. The experiment (Exp.) A1 in Table 
is used as the baseline case where two paths are identical 
(i.e., same propagation delay and bandwidth). Pig. shows 
the RBDs of the considered packet scheduling algorithms 
in this case. One can see that even when two paths are 
identical, the out-of-order packet phenomenon still happens 
for all algorithms (the RBDs take values other than zero). 
























Fig. 4: Simulation topologies 


TABLE I: Simulation parameters 


Parameter 

Value 

Number of simulation runs for each 
algorithm 

20 times 

Simulation time for an experiment 

50 Seconds 

Data packet size 

1000 Bytes 

MSS 

934 Bytes 

ACK option 

Selective ACK 

Queue management 

RED 

Router queue size 

Bandwidth-Delay Product 

Receive buffer 

unlimit 

Bandwidth side 

2 times of bottleneck link band¬ 
width 

Delay side 

1 ms 

Shortest forward-delay path update in¬ 
terval 

every RTT 


Nevertheless, FDPS gets the best performance compared to all 
other algorithms; its RBD sharply concentrates around zero. 

Next, in Exp. A2, the delay of path 2 is increased. The 
RBDs of the considered algorithms are shown in Fig. [6 
The performances of both FIFO and MTCS are degradec 
significantly. Kim-2012 performs much better than FIFO and 
MTCS in this case, but the performance of FDPS is still the 
best. 

Finally, in Exp. A3, we run simulation on two paths with 
different delay and bandwidth (path with lower bandwidth has 
smaller delay). Fig.jTja) and Fig.|7Jb) show the RBDs and RDs 
of the considered algorithms, respectively. We still see that 
Kim-2012 is better than FIFO and MTCS, but FDPS achieves 
the best performance in both metrics. In particular, the packet 
reorder density of FDPS concentrates densely around zero, 
while those of the others spread widely. 

B. Two-path MPTCP with background traffic 

In practice, a MPTCP or TCP fiow in Internet usually 
shares the bandwidth with one or more other fiows in the 
forward and/or backward direction. To evaluate the efficiency 
of a packet scheduling algorithm when MPTCP sends data 


TABLE II: Experiment scenarios 


Experiment 

ID 

Bottleneck link 1 

Bottleneck link 2 

Exp. A1 
(Fig.ga)) 

BW = 4 Mbps, 
delay = 10 ms 

BW = 4 Mbps, 
delay = 10 ms 

Exp. A^2 

(Fig.ga)) 

BW = 4 Mbps, 
delay = 10 ms 

BW = 4 Mbps, 
delay = 30ms 

Exp. A^3 

(Fig.ga)) 

BW = 2 Mbps, 
delay = 10 ms 

BW = 8 Mbps, 
delay = 30ms 

Exp. ^4 

(Fig.gb)) 

BW = 4 Mbps, 
delay = 10 ms 

BW = 4 Mbps, 
delay = 30ms 

Exp. ^5 

(Eig.|^c)) 

forward BW = 4 Mbps, 
forward delay = 10 ms; 
backward BW = 0.5 Mbps, 
backward delay = 15 ms 

forward BW = 8 Mbps, 
forward delay = 30 ms; 
backward BW = 1 Mbps, 
backward delay = 35 ms 


0 . 6 - 

0.4 


Q 
QQ 
^ 0.2 


0.0 


FIFO 




MTCS 


FDPS 


10 20 
Buffer-occupacy (MSS) 


y/-r-n 

50 60 


Eig. 5: RBD of the algorithms in Exp. A1 


packets on such paths, we add one or more regular TCP flows 
to generate background traffic on each path. 

In our fourth experiment, Exp. A4, the path configuration 
is the same as in Exp. A2, but there is background traffic in 
the forward direction as shown in Eig. |^b). The results in 
Eig. show that all algorithms suffer significant degradation 
in performance because the background fiows make the RTT 
estimates less precise (due to large RTT variant) at the sender. 
Both Kim-2012 and MTCS yield poor performance than does 
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Fig. 7: RED (a) and RD (b) of the algorithms in Exp. A3 
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Fig. 8: RED of the algorithms in Exp. A4 


the native FIFO algorithm because both are more sensitive to 
time variant than FIFO. However, FDPS still achieves the best 
performance in buffer occupancy. 

In the fifth experiment, Exp. A5, we use an asymmetric 
topology as shown in Fig. Qc) with background fiows in both 
forward and backward directions. Note that when regular TCP 




Fig. 9: RED of the algorithms in Exp. A5 


fiows send data packets mixing with ACK packets (with small 
packet size), at the intermediate router(s) the ACK packets 
sometimes get to wait until data packets of background fiows 
are forwarded first (this technique is called ACK compression 
|T5|). This might cause the measured RTT values become 
too small or too big, and hence, affect the performances of 
all considered algorithms, especially FIFO, as shown in Fig. 

Nevertheless, FDPS still outperforms the others since it 
estimates only forward-delay differences between paths, thus 
not affected much by the backward paths. 

C Three-path MPTCP 

In the last experiment, we would like to evaluate the 
effectiveness of FDPS on a three-path MPTCP configuration. 
In particular, we run simulation on the topology in which 
paths have diverse values of bandwidth and delay (shown 
in Fig. Ud)). Each path has a regular TCP flow generating 
background traffic. The results are shown in Fig. We can 
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see that, compared to the two-path case, the performances of 
all considered algorithms get worse. It suggests that when the 
number of concurrent paths increases, the reordering problem 
gets more serious. In this case, FDPS again achieves the 
best reordering performance among the considered scheduling 
algorithms. 

















































D. Summary and discussions 


In summary, Table III shows the means (and standard 
deviations) of buffer occupancy for all considered algorithms 
under the above experiments. Our proposed FDPS algorithm 
consistently gets the best buffer occupancy in all experiments, 
which suggests that FDPS is robust under various network 
conditions. 


TABLE III: Mean of buffer occupancy (in MSS) 


Experiment ID 

FIFO 

MTCS 

Kim-2012 

FDPS 

Exp. A1 

4.5±0.16 

4.3±0.18 

2.8±0.08 

2.2±0.02 

Exp. A2 

167.6±0.06 

22.7±0.02 

4.7±0.01 

2.2±0.03 

Exp. A3 

105.5±0.04 

lO.OiO.Ol 

5.4±0.01 

2.6±0.02 

Exp. A4 

272.3±10.8 

19.8±0.42 

16.5±0.78 

8.4±0.06 

Exp. A5 

1682±57.1 

44.6±2.56 

59.4±3.91 

29.2±2.40 

Exp. 3-paths 

790.1±67.2 

68.7±1.96 

32.4±2.02 

24.5±1.14 


Note that in our simulations, we use the unlimited reorder 
buffer size because we would like to capture all data packets at 
the receiver to have the correct RBD and RD measurements. In 
practice, that buffer size is usually finite, and hence, may cause 
packet loss if the out-of-order packet problem gets serious. 
Since FDPS achieves the best reorder buffer occupancy com¬ 
pared to previous solutions, the effect of finite reorder buffer 
size would be minimal if FDPS is used. 

V. Conclusions 

In this paper, we introduce the forward-delay-based packet 
scheduling (FDPS) algorithm for Multipath-TCP to address the 
out-of-order received packet problem occurring when packets 
are transmitted via multiple concurrent paths. The algorithm is 
based on our novel technique of estimating the forward delay 
differences between paths. Our extensive simulation results 
show that our algorithm significantly reduces the reordering 
buffer occupation compared to the previous solutions to this 
problem. 
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